Toward Evaluation of Writing Style: Finding Overly Repetitive Word Use in Student Essays

Automated essay scoring is now an established capability used from elementary school through graduate school for purposes of instruction and assessment. Newer applications provide automated diagnostic feedback about student writing. Feedback includes errors in grammar, usage, and mechanics, comments about writing style, and evaluation of discourse structure. This paper reports on a system that evaluates a characteristic of lower quality essay writing style: repetitious word use. This capability is embedded in a commercial writing assessment application, Criterion. The system uses a machine-learning approach with word-based features to model repetitious word use in an essay. System performance well exceeds several baseline algorithms. Agreement between the system and a single human judge exceeds agreement between two human judges.